Skip to content

DEV-896_2 Frame-clock-driven live plot scrolling and per-plot stroke width - #320

Closed
jyong15 wants to merge 64 commits into
DEV-896from
DEV-896_2
Closed

jyong15 wants to merge 64 commits into
DEV-896from
DEV-896_2

Conversation

@jyong15

@jyong15 jyong15 commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Second part of DEV-896. Hardware testing of #280 on a UHD monitor (one GSR trace at 256 Hz) showed no improvement, and a read-only investigation measured why: the per-sample work #280 removes is under 1 ms of CPU per second in that configuration, and the plot's jank comes from frame cadence, not from the data thread. jchart2d repaints from a 50 ms Swing timer (about 20 fps), and Consensys re-anchored the X viewport to the newest sample's timestamp on every sample, so each frame jumped by whatever Bluetooth burst had arrived since the last one. At UHD each jump covers about twice the pixels.

Stacked on DEV-896 (#280) because it edits the same file. Retarget to master once #280 merges, or rebase if #280 is trimmed.

Changes (ShimmerDriverPC, BasicPlotManagerPC)

Live X viewport frame clock. One javax.swing.Timer per plot manager (FRAME_INTERVAL_MS = 16; 33 is the documented fallback) advances the X window by elapsed time on the EDT and slews it toward the newest sample minus a VIEWPORT_LATENCY_MS = 200 playout buffer, so bursts no longer appear as jumps. The window is not anchored to wall-clock: the plotted X is device time plus a one-time offset, so it drifts and NTP can step the PC clock; the clock follows the data's own rate instead. Catch-up after a stall is asymmetric (up to 2x real time when behind, 10% when ahead), with a 1 s resync snap. The window never passes the newest sample and never scrolls backwards.

  • Anchored fallback. The clock estimates the X rate and sample spacing on the EDT every 500 ms. When X is not advancing at wall-clock rate (DB playback at x2/x4, where X runs 2 to 4x faster) or samples are sparse (below about 4 Hz, e.g. the 2 Hz aggregator plot), it switches to anchoring the window on the newest X each frame, with hysteresis and a two-window vote so a single jump cannot flip it.
  • Multi-device plots. A per-plot newest-X tracker takes the maximum across devices, so the window follows the fastest device without jumping backwards; a genuine backward jump (reconnect, device reset) resyncs after a 500 ms confirmation.
  • Lifecycle. Started by a throttled request from the data thread when live time-axis data arrives on a displayed chart. Stopped after 2 s without data, on pause, when the chart is hidden or its window iconified, on a non-time axis, on clear, and on removeAllSignals() (the frame-close path). jchart2d's own repaint timer is backed off to 250 ms while the clock runs and restored on stop, so there is exactly one paint per frame and an idle app does not repaint at 60 fps. One RangePolicyFixedViewport with two swapped Range objects: no allocation per frame.
  • Opt-in via setLiveViewportFrameClockEnabled(true); nothing changes for consumers that do not call it. The per-sample setXAxisRangeBasedOnXDuration() stays for explicit callers.

Per-plot stroke width. New setTraceLineThickness(float), read by createNormalTrace and setTraceLineStyle. DEFAULT_LINE_THICKNESS stays 2, so reports, NeuroLynQ, event markers and calibration are unchanged. Consensys live plots set 1 px (companion PR): at identity transform a 2560-point UHD paint measured 0.4 ms at 1 px against 3.7 ms at 2 px (drawPolyline); at uiScale 2 the difference is 3.5 vs 4.0 ms.

mCurrentXValue is now volatile.

Companion

Shimmer-Advance-API DEV-896_2 enables the clock and the 1 px stroke for Consensys live plots and adds a 1.2 s trace margin (LIVE_VIEWPORT_TRACE_MARGIN_MS = FRAME_CLOCK_RESYNC_MS + VIEWPORT_LATENCY_MS, sized for post-stall catch-up) so the trailing window never shows a blank strip. ConsensysApps needs no change. Merge order: this repo first (#280 then this), then the Shimmer-Advance-API PR, since its CI clones this repo's default branch.

Testing

  • New API_00029_LiveViewportFrameClockTest (Swing-free, synthetic clock): monotonic window, bounded lag, 600 ms stall recovery within 1 s, 300 ms EDT hiccup, a device 1.5 s behind, a device reset, 4x playback without snaps, 2 Hz data without frozen runs. 7/7 pass.
  • ShimmerDriverPC ./gradlew build: 58 + 177 tests, 0 failures. ShimmerDriverAdvancedPC, ConsensysApi and Consensys compile against this branch.
  • Headless harness with a real Swing timer: X step per 16 ms tick 16.2 ± 0.7 ms, versus ± 34 ms with 80% zero-steps on the old path. Playback at 2.6x / 3.9x / 5.2x: forward snaps 17 / 50 / 82 per 25 s before the anchored fallback, 0 after.
  • Two adversarial review rounds; all findings fixed before opening.
  • Not yet run in the real app. Needs the UHD streaming check, DB playback at x2/x4, and EDT load with several plots open (switch FRAME_INTERVAL_MS to 33 if the EDT saturates; estimates put that at about 4 plots with 8 traces each).

🤖 Generated with Claude Code

https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML


Generated by Claude Code

claude and others added 30 commits July 16, 2026 10:33
ShimmerBluetooth.IOThread and LiteProtocol.IOThread ran tight while(!stop)
loops with no sleep, calling processNextInstruction()/bytesAvailableToBeRead()
as fast as possible, pegging a CPU core per connected device even when idle.
Track whether an iteration did useful work (instruction processed, actively
streaming, or bytes read) and only sleep briefly (2ms) when it didn't -
latency of the active/streaming paths is unchanged since sleep is skipped
whenever there's real work to do.

ShimmerBluetooth.ProcessingThread spun on mABQPacketByeArray.isEmpty(); since
the queue is an ArrayBlockingQueue, swap to a blocking poll(100ms) instead so
the thread parks while idle rather than busy-checking emptiness.

Also replace two teardown busy-waits with bounded Thread.join() calls so
closing a connection doesn't spin a core while waiting for the reader
threads to exit: ShimmerPC.closeConnection() (mIOThread + mPThread) and
LiteProtocol.stopIoThread() (mIOThread).

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JjgNkS6wtPkQUdK3KdHVdN
…un(), interrupt-exit loops

Review findings on the idle-sleep changeset:
- LiteProtocol.IOThread.run() was declared synchronized; Thread.join() locks
  the same Thread instance, so external stopIoThread() callers blocked on
  monitor entry with no timeout until the loop exited, defeating the bounded
  join. run() is no longer synchronized.
- stopIoThread()/closeConnection() can be reached synchronously from the
  reader thread itself (killConnection from IOThread's catch; connectionLost
  on serial errors), where join() can never succeed and just burns the full
  2s timeout. Skip the join when running on that thread.
- Reader loops now also exit on thread interrupt: previously an interrupt
  without stop=true would make poll()/sleep() throw immediately every
  iteration - a latent busy-spin.

Co-Authored-By: Claude Fable 5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01JjgNkS6wtPkQUdK3KdHVdN
processNextInstruction()'s no-instruction branch already waits 50ms
(threadSleep(50)) when idle (no instruction pending, not streaming, no
bytes available). The new outer idle-sleep in IOThread.run() added a
further Thread.sleep(2) whenever didWork stayed false, so the fully
idle case waited 52ms instead of 50ms.

Fix: have that branch report true (i.e. "waited") instead of false, so
run()'s outer idle-sleep is skipped for the iteration that already
paced itself. This is the minimal change - it keeps the pre-existing
50ms idle cadence exactly as before this PR, rather than replacing it
with the new 2ms cadence (a bigger behavioural change) or leaving the
two sleeps stacked. The 50ms sleep only fires when there is no
instruction on the stack at all, so it has no bearing on pacing
retries for an in-flight instruction - removing it in favour of the
2ms sleep was rejected because that would make idle polling 25x more
frequent than before.

Applied identically to ShimmerBluetooth.java and LiteProtocol.java,
with matching javadoc updates on processNextInstruction().

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
…cessingThread self-join, volatile stop

Round-2 review findings on the bounded-join teardown from the previous
commit:

- ShimmerPC.closeConnection() and ShimmerGRPC.closeConnection() are
  unsynchronized and can run concurrently (connectionLost() on the
  IOThread itself vs. a user-triggered disconnect() on another
  thread). Both dereferenced mIOThread/mPThread after only a
  non-atomic null check, so a concurrent caller nulling the field out
  mid-method could turn the passed check into an NPE - swallowed by
  the outer catch(Exception), skipping purgePort/closePort and leaking
  the COM port. Fixed by snapshotting mIOThread/mPThread into locals
  once and operating on the snapshot throughout.
- Added the same self-join guard used for mIOThread to mPThread in
  both classes: buildAndSendMsg() runs on the ProcessingThread and
  dispatches to app listeners, so a listener calling disconnect()
  would otherwise self-join and burn the full 2s timeout.
- After each bounded join(2000), if the thread is still alive, log a
  warning and call interrupt() so the thread's own interrupt checks
  can finish tearing it down, instead of silently closing the port
  under a still-live reader. Applied to mIOThread/mPThread in
  ShimmerPC.closeConnection() and ShimmerGRPC.closeConnection(), and
  to mIOThread in LiteProtocol.stopIoThread().
- ShimmerGRPC.closeConnection() previously did an unbounded join()
  with no self-join guard; converted to the same bounded
  snapshot+guard+alive-check pattern, keeping its existing interrupt()
  calls.
- Made the `stop` flags volatile (ShimmerBluetooth.IOThread,
  ShimmerBluetooth.ProcessingThread, LiteProtocol.IOThread) so a
  writer on one thread is visibly seen by the reader thread without
  relying on incidental synchronization elsewhere.
- Softened the processNextInstruction() idle-branch comment in
  ShimmerBluetooth.java: getInstruction() returning null doesn't only
  mean "no instruction pending" - it also happens right after removing
  a leading null placeholder, with a real instruction possibly next in
  the list. No behavioural change, comment only.

Re-verified: streaming path in both IOThread.run() loops still never
sleeps, every idle path still sleeps (no spin), and closeConnection()
still reaches purgePort/closePort on every path.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
…errupt before join, tidy-ups

Copilot review findings on PR #279:

- ShimmerBluetooth.java and LiteProtocol.java: the idle-branch sleep
  in processNextInstruction() used threadSleep(50), whose shared
  implementation swallows InterruptedException without restoring the
  interrupt flag - defeating run()'s
  !Thread.currentThread().isInterrupted() exit check. Replaced with a
  local try/Thread.sleep(50)/catch that re-sets the flag via
  Thread.currentThread().interrupt(). Left every other threadSleep()
  call in both files untouched.
- ShimmerBluetooth.java IOThread.run(): fixed inconsistent indentation
  in the InShimmerTest branch (didWork assignment and the readBytes
  block were not nested under the bytesAvailableToBeRead() guard).
  Indentation only - control flow verified unchanged.
- ShimmerPC.java: fixed a duplicated "before before" in a comment.
- ShimmerPC.closeConnection(), ShimmerGRPC.closeConnection(), and
  LiteProtocol.stopIoThread(): interrupt() is now called before the
  bounded join(2000) (still only in the non-self-join branch) instead
  of only after a failed join. Since run() now exits on
  Thread.currentThread().isInterrupted() and any idle Thread.sleep()/
  queue poll() wakes immediately on interrupt, this makes shutdown
  faster and more deterministic. The post-join "still alive" branch
  now just logs a warning instead of interrupting a second time.
- ShimmerGRPC.closeConnection(): moved the pre-existing interrupt()
  calls (for both mIOThread and mPThread) inside the
  Thread.currentThread() != t guard - previously they ran
  unconditionally and would mark the *current* thread interrupted
  when closeConnection() runs on the IOThread/ProcessingThread itself
  (the self-close case), which is neither needed nor correct.

Safety check for interrupting a thread blocked in a serial read
(requested before making the interrupt-before-join change): ShimmerPC
reads via ByteCommunicationJSSC, which calls JSSC's
SerialPort.readBytes(count, timeout) - a native, timeout-bounded
JNI call that is not interruptible and never throws
InterruptedException; Thread.interrupt() on a thread blocked there
only sets the interrupt flag, observed once the call returns (by
timeout or data) at the top of run() or in a subsequent
Thread.sleep()/queue.poll(). bytesAvailableToBeRead()/readBytes()/
writeBytes() only catch SerialPortException/SerialPortTimeoutException/
NullPointerException, none of which interrupt() can trigger, so no new
connectionLost() -> closeConnection() recursion is introduced.
ShimmerGRPC reads via a ThreadSafeByteFifoBuffer whose read() does
declare InterruptedException and is already caught locally (printed
and treated as "no data"), so it is safe by construction. No path was
found where the interrupt flag causes a new failure mode; the change
was made in all three teardown sites as requested.

Re-verified: both IOThread.run() loops still never sleep on the
streaming path, no path can spin without sleeping, and
closeConnection() still reaches purgePort/closePort on every path.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_015q29AEJZbwa7aVYeAj61Py
- LiteProtocol.stopIoThread(): snapshot mIOThread into a local and use it
  for stop/interrupt/join/isAlive, so a concurrent teardown that nulls the
  field after the null check can no longer cause an NPE. Only clear the
  field if it still points at the thread that was stopped, so a newly
  started IOThread is not clobbered.
- ShimmerPC/ShimmerGRPC.closeConnection(): move the ProcessingThread
  (mPThread) teardown out of the `if (ioThread != null)` block into its own
  sibling block, so it still runs when another caller has already nulled
  mIOThread while the ProcessingThread is alive. The logic of each block
  is otherwise unchanged.

The merge of origin/master is in the preceding merge commit.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
F1: closeConnection() closes the serial port after a bounded 2 s join even
if the IOThread is still blocked in a non-interruptible jssc call. When that
call then failed on the closed port (or NPE'd on the nulled
mByteCommunication), the ShimmerPC I/O helpers called connectionLost(),
setting CONNECTION_LOST after a deliberate disconnect, which
AbstractConsensysApi answers with an auto-reconnect in NeuroLynQ. The catch
sites in bytesAvailableToBeRead/availableBytes/writeBytes/readBytes, and
ShimmerGRPC.writeBytes, now skip connectionLost() when running on an
IOThread whose stop flag is set. Both closeConnection()s also clear
mIOThread/mPThread only if they still hold the thread that was stopped, as
LiteProtocol.stopIoThread() already does, so a late closeConnection() cannot
clobber a reconnect's new threads.

F2: LiteProtocol.IOThread.run() no longer calls killConnection() when stop
is already set: the teardown interrupt surfacing from txBytes as
ERR_WRITING_DATA caused a spurious error event and a re-entrant disconnect.

Test: API_00028_IoThreadTeardownTest (ShimmerDriverPC) pins that idle
IOThread/ProcessingThread use <300 ms CPU per second, that disconnect()
stops both within 1500 ms, and that a write blocked past the join and then
failing on the closed port leaves the device DISCONNECTED (fails without F1).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
Live time-axis plots moved their X window from the data thread after every
sample (a PlotCustomFeature calling setXAxisRangeBasedOnXDuration(), which
also allocated a new RangePolicyFixedViewport 256 times a second). The
window's right edge was the newest sample's X, so it advanced in Bluetooth
bursts of 10-30 samples, and Chart2D's 50 ms repaint timer showed each burst
as a jump - worst at UHD, where a jump covers twice the pixels.

BasicPlotManagerPC now has an opt-in frame clock
(setLiveViewportFrameClockEnabled): one javax.swing.Timer per plot manager,
FRAME_INTERVAL_MS = 16 (33 is the documented fallback if 60 fps is too heavy
on the EDT). Each tick, on the EDT, advances the window's right edge by the
real time elapsed since the previous tick and repaints the chart directly, so
each frame shows exactly one even step. While it runs, Chart2D's own repaint
timer is backed off to 250 ms (restored on stop) so it does not add a second,
unsynchronised stream of paints.

Latency design: the right edge is a playout clock that trails "newest X +
time since that sample arrived" by VIEWPORT_LATENCY_MS = 200, which absorbs
the bursts and the IOThread -> callback -> plot-thread hops. It is steered
toward that target by 5% of the error per tick, each correction capped at
10% of the tick's elapsed time, snaps when more than 1 s off, never runs past
the newest sample (a stall freezes the window, as before) and never scrolls
backwards. It is deliberately not anchored to System.currentTimeMillis():
System_Timestamp_Plot is the device clock offset once onto the PC epoch at
the first packet, so it carries that packet's latency and drifts with the
device crystal, and the wall clock can be stepped by NTP; System.nanoTime()
plus the newest sample needs no common epoch. Points newer than the right
edge are clipped by jchart2d, not drawn into the margin.

Lifecycle (DEV-895/DEV-717): the clock starts when live time-axis samples
arrive into a showing chart, and stops itself when none has arrived for 2 s,
the plot is paused, the chart is not showing, the X axis is not the time
axis, or the data was cleared; removeAllSignals() (the frame-close path)
stops it immediately. On stop the window is left ending at the newest
sample. A stopped Swing Timer is not referenced by the TimerQueue, so it
does not pin the frame. The tick takes only the chart monitor, never the
list monitors, so the list -> chart lock order is kept. mCurrentXValue is
now volatile; the data thread's per-sample cost is a nanoTime() read and a
volatile write.

setXAxisRangeBasedOnXDuration() and its sampling-rate variant are unchanged
for explicit callers (ConsensysGQ's right-aligned plots, PlotEventMarkers);
frequency/FFT and value X axes never run the clock.

VISIBLE CHANGE: DEFAULT_LINE_THICKNESS is now 1 (was 2), so live traces are
thinner. A full paint of 2560 points into a 3600x1800 chart measured 9.2 ms
at 2 px against 2.3 ms at 1 px. UtilChart2D's separate constant is left at 2:
only QuickPlot uses it, not live traces.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
…device X

Follow-up to 2e22633 after review.

- Catch-up: below the 1 s snap threshold the playout clock could only speed
  up by 10% of real time, so a 600-1100 ms stall or a 500 ms forward X jump
  left the newest data hidden for 3-8 s, and the lag could outgrow the 0.4 s
  trace margin. The correction is now asymmetric: slowing down and errors
  under 50 ms keep the 10% cap and 5% gain; from 50 ms to 200 ms behind, the
  cap ramps linearly to 100% (2x real-time scrolling) and the gain to 15%.
  Simulated recovery to <260 ms hidden: 300/600/900/1100 ms stalls 68/376/
  668/836 ms, +500/+900 ms X jumps 576/976 ms; no backward steps, no
  overshoot, max step 32 ms per 16 ms tick. The 1 s snap is unchanged.

- Stroke: revert DEFAULT_LINE_THICKNESS to 2, so reports, GQ/NeuroLynQ,
  PlotEventMarkers, PanelCalibrateShimmer and every other plot keep 2 px.
  New per-manager setTraceLineThickness(float), defaulting to
  DEFAULT_LINE_THICKNESS, is used by createNormalTrace() and by
  setTraceLineStyle() for CONTINUOUS/INDIVIDUAL_POINTS (which previously
  re-applied the static). The comment now states what jchart2d does
  (TracePainterPolyline, i.e. Graphics.drawPolyline) and the measured cost
  of a 2560-point UHD paint: ~0.4 vs ~3.7 ms for 1 vs 2 px at identity
  transform, ~3.5 vs ~4.0 ms at uiScale 2.

- First frame: startFrameClockOnEdt() now runs one tick synchronously, so
  a chart shown again does not paint its stale window for the first frame.

- Multi-device plots: mCurrentXValue alternates between the devices'
  timestamps, so with a >=1 s offset the snap branch made the window jump
  back and forth, and with smaller offsets it followed the slower device.
  The clock now uses mFrameClockNewestX (and the arrival time of that X),
  kept by the data thread as max(X); it restarts from a lower X only on a
  genuine backward jump: more than 1 s behind with no sample near the newest
  for 500 ms (without that confirmation a 1300 ms device offset still
  flipped it on every sample). Reset to 0 with mCurrentXValue in
  removeAllSignals() and clearAllDataBuffer(). Idle detection still uses
  any device's last sample. mCurrentXValue itself is unchanged. Simulated
  0-1300 ms offsets: zero backward steps, hidden lag vs the faster device
  mean 172 ms (144-201); a -5 s jump re-syncs with one snap.

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
Round-2 review follow-up to 5aeba04.

The playout clock assumes X advances ~1 ms per wall-clock ms and that new
X arrives well within VIEWPORT_LATENCY_MS. Two real cases break that:
- Consensys DB playback runs recorded data through the same live plots;
  PlaybackSession's timer period is truncated, so X runs ~1.3x at "x1" and
  2-4x+ at x2/x4. The 2x catch-up cap could not keep up, the 1 s resync
  snapped forward 17-82 times per 25 s and a blank strip opened at the left.
- Low-rate series (the 2 Hz fusion-response plot, any device < 5 Hz)
  froze at the newest-sample clamp, then caught up at 2x (40% frozen ticks
  at 2 Hz).

computeLiveViewportEnd() now estimates, on the EDT, the X rate
(dNewestX/dArrival) and the spacing between new X values over windows of
>= 500 ms of arrivals (so a stall followed by buffered data still reads as
~1x), EWMA-smoothed (alpha 0.5). If the rate is > 1.5 or < 0.67, or the
spacing > 1.2 x VIEWPORT_LATENCY_MS, in both the window and the EWMA for
two windows in a row, it switches to anchored mode: the right edge is the
newest X, as before DEV-896 but still one update and one paint per tick.
When anchored for fast X, each tick may advance at most 1.5x the estimated
rate, so two playback deliveries landing in one 16 ms frame are spread over
the following frames instead of drawing as one double step. It switches back
when the rate is within 0.83-1.2 and the spacing < 0.75 x latency for two
windows, then holds the window still until the playout target catches up
(~110-130 ms) instead of stepping back. The estimate resets when the clock
stops.
The spacing threshold is 1.2x the latency, not 0.75x: a 5 Hz series still
scrolls evenly in playout mode (0% frozen) and would only get jumpier
anchored.

Sims (25 s each): playback 2.6/3.9/5.2x go from 17/50/82 forward snaps to 0,
with lag behind the newest X <= 37 ms (trace margin 400). At 1/2/3/4 Hz the
longest frozen run is one sample interval, and the only fast ticks are the
anchored jumps themselves. The stall, X-jump and two-device tables from the
last round are unchanged.

Also:
- onFrameClockTick() stops without snapping when the X axis is no longer
  the time axis, so a value/frequency axis is not given a time window.
- The clock also stops when the chart's top-level Frame is iconified (an
  iconified JFrame's components still report isShowing()), and is not
  started while it is.
- New API_00029_LiveViewportFrameClockTest (Swing-free, synthetic
  nanoTime): steady stream, stall catch-up, long EDT tick, far-behind
  second device, genuine backward jump, X rate 4x (no step > 100 ms, lag
  within the trace margin), 2 Hz data (no frozen run longer than one sample
  interval).

Co-Authored-By: Claude Fable 5.1 <[email protected]>
Claude-Session: https://claude.ai/code/session_01FkxGgMBTsQEZ36RNU3khML
DEV-895 Stop per-device reader threads busy-spinning when idle
…n tests

The simulator connect tests called connect() and then slept a fixed 3-5 s
before asserting on the device. isConnected() turns true before the
post-connect config read (infomem, calibration) runs, so on a loaded CI
runner the read could miss the window and assertions such as
getTrialDurationEstimatedInSecs() failed - seen in Shimmer-Advance-API
run 37567210311, which passed on rerun.

Add ByteCommunicationTestUtils.connectAndWaitUntilInitialised(), which
waits until isInitialised() and the CONNECTED state (30 s ceiling, fails
with a message on timeout), and use it for all 22 connect waits. The
TaskCompletionSources it replaces were never completed, so they were only
ever sleeps. Also disconnect the device in @after so earlier tests' IO
threads and status timers no longer run on into later tests.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Lets the Shimmer3R SD log parser set the low-noise accel (LSM6DSV)
range from the file header, as setLSM6DSVGyroRange does for the gyro.
mSensorLSM6DSV is private, and ShimmerObject.setAccelRange sets the
wide-range accel on Shimmer3R, so there was no way to reach it.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…_accel_range

DEV-1163 Add ShimmerObject.setLSM6DSVAccelRange for the Shimmer3R SD log parser
Wait for initialisation instead of a fixed sleep in byte-communication tests
Process.onExit() is Java 9+. ShimmerJavaClass.jar is built as Java 8
bytecode for MATLAB, and source/target 1.8 compiled this call without
complaint, so starting the BLE gRPC server from MATLAB on Java 8 would
throw NoSuchMethodError. The jar currently in Shimmer-MATLAB-ID/libs
(built 2025-11-24) contains the call.

Wait on the process in a daemon thread instead, which logs the same
exit message. Found by building the jar with --release 8.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Actions -> "Release Java API" releases the version already in the
build files (bumped by the usual uprev PR; master is protected):
- tests and builds as gradle.yml does
- publishes shimmerdriver, shimmerbluetoothmanager and shimmerdriverpc
  in one gradlew publish, so they cannot drift apart again
- tags API_v<version> and creates the GitHub Release with the jars,
  not marked Latest
- builds ShimmerJavaClass.jar as Java 8 bytecode on its own runner and
  opens a PR replacing Shimmer-MATLAB-ID/libs/ShimmerJavaClass.jar
  (needs the MATLAB_REPO_TOKEN secret)

ShimmerDriverPC gains -PmatlabJava8, which compiles all three projects
with --release 8 in place of hand-editing three build files to 1.8.

Publishing URLs move from ShimmerEngineering, now a user account, to
ShimmerResearch. AGENTS.md documents the release steps.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The version in the build files is normally one that has already been
released until the next uprev, so stopping there made every PR dry run
end before the build it is meant to test. Dry runs now warn; a real
release still refuses.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The lns0_16_11_w_sd_bt_streaming connect hangs on Linux CI and not on
Windows, so the CI log is the only evidence of where the IO thread is
stuck. Print every thread's full stack, the monitors it holds, the lock
it waits on and any JVM-detected deadlock before failing.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The connect sequence unlocks the instruction stack and then starts the
read-status timer with no delay, so the timer's first readStatusTask
(device lock, then instruction-list lock) runs just as the IOThread takes
the first post-connect instruction in getInstruction() (list lock, then
device lock). When they meet, both block for good and the connect never
finishes. The thread dump from the previous commit shows that cycle on
Linux CI in lns0_16_11_w_sd_bt_streaming, and locally in
API_0000X_ByteCommunicationShimmer3.

The driver is deliberately left unchanged. The simulator tests now build
their ShimmerPC through ByteCommunicationTestUtils.newShimmerPCWithoutStatusTimer(),
which overrides startTimerReadStatus() to do nothing, so nothing else
touches the instruction list while the IOThread connects. The tests lose
only the 5 s status poll; the connect sequence still reads the status
once.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1198 Fix the ShimmerDriverPC connect test hang on Linux CI
API_0000X_ByteCommunicationShimmer3lns0_16_11_w_sd_only, _w_sync_master
and _w_sync_slave were excluded from the test task in Feb 2026 as failing
on CI. Their CI logs have expired, so the cause cannot be confirmed, but
it fits the read-status timer deadlock the previous commit avoids: these
classes assert values from the post-connect config read (SD log sync,
Bluetooth disabled, sync nodes, trial duration), which a stalled connect
never reaches. They now build their device through
newShimmerPCWithoutStatusTimer() like the other simulator classes, and
run again.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The version no longer lives in the source. The three publishing blocks
read -PreleaseVersion (falling back to 'local'), and Release Java API
passes it: the version typed in the Run form, or by default the next
patch after the last API_v release (0.11.8_beta -> 0.11.9_beta). The
API_v tags record what was released, so a release needs no uprev PR
and nothing has to be committed to the protected master.

Checked locally: with -PreleaseVersion=0.11.9_beta all three POMs and
their dependencies on each other are 0.11.9_beta; without it, 'local'.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
publish failed on shimmerdriverpc with "multiple artifacts with the
identical extension and classifier ('jar', 'null')": the Shadow plugin
adds ShimmerJavaClass.jar (empty classifier) to components.java, and
Gradle 8.14 (DEV-1013) rejects the clash. The PR dry run skips publish,
so the first real release would have failed there after publishing the
other two. Skip the shadow variant, which publishes the plain jar with
its dependencies in the POM, as 0.11.7_beta was.

Without -PreleaseVersion the version is 'local', and a hand-run
`gradlew publish` would push that to GitHub Packages, where it can
never be overwritten. Publishing to a remote repository now refuses
'local'; publishToMavenLocal is unaffected.

Add job timeouts so a hung Gradle daemon cannot hold a runner for the
default 6 hours.

Checked locally (JDK 11): publishToMavenLocal -PreleaseVersion=0.0.0_test
now succeeds, all three POMs and their cross-dependencies read
0.0.0_test, and shimmerdriverpc is the 140 KB thin jar. Publishing to a
file repository without -PreleaseVersion fails for all three.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1115 Fix shimmerdriverpc publish; refuse a 'local' publish
JongChern and others added 23 commits October 21, 2026 16:01
…Classic, and a release gate

Hardware-in-the-loop tests of ShimmerDriver/ShimmerDriverPC on a PC with paired units, through
ShimmerPC with its timers on, as Consensys and MATLAB run it.

- JavaHilScenariosTest (hilTest task, skipped without -Phil.port, left out of the normal test
  task): streamRest, midStreamSet, reconnectLoop and inStream. inStream requests each in-stream
  reply (status, directory, battery) and then overlaps SET_RWC with status and battery requests,
  the DEV-1262 case. Every API call runs under a watchdog that dumps every thread with its locks.
- HilShimmerPC, a test-only subclass, counts in-stream replies by type, decoded statuses and
  battery readings, and captures every driver log line, without changing what the driver does.
  The configure step does not write the calibration dump back, so runs leave the units' stored
  calibration alone.
- StreamAnalysis is the Android HIL's class (same rules on both); its tests run in CI.
- tools/hil/run-java-hil.ps1 finds each unit's COM port from its MAC, builds once, runs each
  scenario in its own Gradle run under a time limit, checks every report came back, and with
  -Report posts the commit status java-hil (full default run, clean pushed commit only).
- Release Java API stops unless java-hil passed on the commit, or hil_waiver gives a Jira key;
  the release notes record which.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ink Windows can't open

midStreamSet and inStream now start SD logging as Consensys does (DEV-436):
stop streaming, start logging on the quiet link, then stream again. SD
logging is still stopped mid-stream. Consensys never writes the RTC
mid-stream, so the scenarios no longer do either; that is what caused
DEV-1245's out-of-order timestamps on the Shimmer3R.

The stream is judged per stretch between the deliberate restarts. Loss
within 1 s of an SD logging stop is reported but does not fail, because
the unit pauses while it closes its file. Loss near a battery read fails,
and loss anywhere else is held to the limit.

A connect whose Bluetooth link can't be opened (CONNECTION_FAILED; jssc
reports "Port busy") is tried once more, and connectRetries in the report
counts those retries.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The pattern expected the command as one word, but the driver logs it as
"[0x72] GET_STATUS_COMMAND". So every commandTimeouts count was 0, and the
"no command timeouts" checks could never fail. Checked against the bench
logs so far: it now finds all 14 timeouts in them.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The accel check was skipped when the mean was NaN, so a stream with no
calibrated accel passed. A Shimmer3 (3E36, LogAndStream v1.1.7) did exactly
that: all 7655 samples NaN after the driver read its calibration dump.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…trying it

Every "Port busy" the HIL ever met was another program holding the COM
port, never a flaky first connect. Two HILs shared a unit, and that sharing
produced the misleading DEV-1276. A retry only lets the run go on while the
unit is shared, so its results mean nothing.

So a connect is no longer retried. When it fails, the report says why:
- the port is open in another program (jssc "Port busy"), which the HIL
  now sees because HilShimmerPC records the messages the driver prints
  with consolePrintException;
- the link could not be opened for another reason;
- the unit did not initialise in time.

The README now says that nothing else may use the units during a run, and
why a BLE host is the dangerous case.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
The connect sequence reads the unit's status part way through. On that
reply, ShimmerPC.eventLogAndStreamStatusChanged() moved the state from
CONNECTING to CONNECTED, and ShimmerDeviceCallbackAdapter marks every
CONNECTED as initialised. So an app waiting for CONNECTED and initialised
was told the unit was ready while the battery read, the calibration dump
and the inquiry were still queued.

Whatever the app did next went out in among that sequence:
- streaming started part way through it, and a reply from the connect
  sequence landed in the stream, which then went silent;
- a configure cut the calibration dump short, which left a Shimmer3's
  calibrated accel NaN in every sample (7 of 9 runs on the HIL).

The sequence's own end, inquiryDone() and then isReadyForStreaming(), already
moves CONNECTING to CONNECTED and sends NOTIFICATION_SHIMMER_FULLY_INITIALIZED.
A status reply now leaves CONNECTING to it.

The Shimmer3R simulator answered INQUIRY with the Shimmer3's 8-byte settings,
which never parse as a Shimmer3R's 11. Its tests had only ever passed on the
early CONNECTED, before the inquiry timed out. It now answers with settings
that match its own InfoMem.

API_00031 connects to the Shimmer3 simulator, whose sequence reads the status
before the inquiry. It checks that CONNECTED comes only after the sequence's
end, with nothing but the RTC write still queued. It fails without the fix.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
After STOP_LOGGING_ONLY_COMMAND the unit stops sending data packets for about
100 ms while it closes its SD file. A command sent in that pause is answered
at once, so its ACK comes straight after the previous ACK's CRC, with no data
packet before it.

processPacket() only looks for an ACK after a data packet, so it threw the
reply away. The command then timed out, and while streaming the timeout
clears every queued instruction, so whatever the user asked for next
(including Stop Streaming) was silently dropped. The Java HIL found it on a
Shimmer3 and a Shimmer3R: up to 4 of 5 status and battery requests sent
after a logging stop were lost.

processWhileStreaming() now looks for an ACK at the very start of the buffer
first. It takes one only while a command waits for its ACK, and only when the
bytes after it are what that command expects:
- for a GET command, its own in-stream reply (FF 8A, then its response opcode);
- for a SET command, a bare ACK whose link CRC checks out.
So a 0xFF data byte the parser resyncs onto is not taken for one. It looks at
no more than 3 bytes, fewer than processPacket() needs, so nothing else
changes. Without a link CRC the case can't arise: the previous ACK isn't
processed until the stream resumes, so the next command never goes out during
the pause.

API_00032 feeds the parser the bytes a logging stop produces, then a status
reply, a battery reply, or a SET command's bare ACK in the pause, with 1- and
2-byte CRCs. Without the fix, all six of those cases fail. Two more check that
an ACK at the start of the buffer is not taken for a different command.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
ShimmerBluetooth guards its instruction list with two locks: the device (its
synchronized methods) and the list itself. Every accessor takes them device
first, list second. getInstruction(), which the IO thread calls for the next
instruction, took the list and then called the synchronized
getListofInstructions(), so it took them the other way round.

So the IO thread could hold the list while waiting for the device, just as a
timer held the device while waiting for the list:
- the ACK/response timeout, calling clearAllInstructions() while streaming;
- the read-status timer, queuing GET_STATUS_COMMAND.
Both then waited for good, and so did every later API call. The Android HIL
caught it on a Shimmer3, where stopSDLogging() never returned. It was also
the Linux CI hang in DEV-1198, which #325 worked around in the tests only.

getInstruction() now reads the head of the list directly, so the IO thread
never needs the device while it holds the list. This is Mas's suggestion
from her review of #326.

API_00033 reproduces the deadlock deterministically. A "timer" thread holds
the device the way every synchronized accessor does while the IO thread's
real getInstruction() runs; the timer then queues an instruction, which
needs the list. Without the fix, the JVM's deadlock detector finds the IO
thread blocked in getListofInstructions() and the timer blocked in
writeInstruction(), the same stacks as the hardware hang. A second test
connects five times with the read-status timer left on, as the product
runs.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
While a unit streams, four app threads call readBattery() and
readStatusLogAndStream() and check the queue length, in tight loops for 60 s,
keeping the queue under 8. Each of those calls locks the device and then the
instruction list, so one is nearly always under way while the IO thread,
holding the list, fetches the next instruction. That is the window DEV-1266's
lock inversion needs. No SD logging is involved.

The scenario fails if the calls and replies stop for 10 s, or if the JVM
finds a deadlock. Either way the events get every thread's stack, and the
unit is left alone.

On a Shimmer3R (v1.1.21) without the DEV-1266 fix (#334), it deadlocked
within 0.3 s in both runs. With the fix, two 60-second runs were clean, with
about 670 calls and 0 timeouts each.

It runs by default, so the release gate includes it. Until #334 merges, the
gate fails on a deadlock.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
A Shimmer3 (LogAndStream v1.1.7) sometimes gives up one sample slot to
answer GET_VBATT mid-stream: packet n, the battery reply, then packet n+2.
That was 4 skips in 33 battery reads. It's the unit, not the host: there
were no parser errors, and builds with and without the driver fixes skip at
the same rate. Filed for the firmware as DEV-1297. The check charged it as
packet loss near a battery read, so midStreamSet failed on the Shimmer3
about one run in three.

A gap near a battery read is now excused only when it is exactly one sample
and the packet after it arrived within 25 ms of a decoded battery reply.
Such a gap is reported, as skippedAtBatteryReplies plus an event naming
DEV-1297. Anything else near a battery read still fails. A packet the driver
dropped would leave stray bytes, which checkDriverLog() fails on anyway.
HilShimmerPC now records when each battery reply is decoded.

Checked on 3E36: 4 midStreamSet runs all passed. Two of them reported one
skip at a battery reply, and there was no other loss near any read.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
DEV-1269: Java HIL v1, the driver on real Shimmer3/3R over Classic, and a release gate
…ter_pause

DEV-1291: Take an in-stream ACK that comes with no data packet before it
…nect_sequence

DEV-1275: Report a Shimmer ready only once its connect sequence has ended
…t_lock_order

DEV-1266: Take the instruction list's two locks in one order
… end

DEV-1275 was only caught indirectly: a stream lost in reconnectLoop, or NaN
accel when a configure cut the calibration dump short. The HIL now checks it
directly. For each connect it notes whether the first CONNECTED state came
before NOTIFICATION_SHIMMER_FULLY_INITIALIZED, which the driver sends at the
end of its connect sequence. The report counts these as
connectedBeforeReady, and any one fails the scenario.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…check

DEV-1269: Java HIL checks that every connect reports ready only at the end (DEV-1275)
Three checks the C# and shimmer-logandstream HILs already run on the same bench:

- rollover: streams 10 minutes, past the 512 s wrap of the unit's 3-byte timestamp counter, and
  fails on streamRest's checks or if the raw counter never wrapped. TimestampWraps counts the wraps
  from the TIMESTAMP channel's raw ticks, which HilSession now records with each sample.
- idleStart: leaves the link idle 60 s with the driver's timers on, then fails if the link was lost
  or a command timed out during the wait, if streaming starts more than 500 ms after
  startStreaming(), or on the stream checks.
- samplingRates: writes 51.2, 256 and 512 Hz in turn the Consensys way, streams 5 s at each and
  checks the rate, then writes the unit's own rate back.

run-java-hil.ps1 runs all eight scenarios by default, so -Report requires the new ones too.
TimestampWrapsTest runs in CI. No driver code changes.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Every reply path retires the command in flight with removeInstruction(0), taking the head of the
instruction list to be the command answered. Two things broke that around a stop:

- operationPrepare() cleared the list while the STOP was still in flight. The stop handling, which
  sleeps 400 ms on the first byte after the STOP, then removed the head in the STOP's place: the first
  SET_INFOMEM of a configuration started straight after stopStreaming(), the chunk that holds the
  sampling rate. The unit kept its old rate. operationPrepare() now waits, for up to 3 s, until a
  command already in flight has been answered and retired, then clears the list. mTransactionCompleted
  is volatile, so the waiting thread sees it.
- The stop handling then let the same byte go on to the ACK processing. When that byte was the unit's
  ACK, it retired the next instruction as well, a command queued after the stop that was never sent.
  The stop handling now returns once the stop is handled, and logs when it was answered by its ACK.

API_00031_ConfigureStraightAfterStopTest runs the IOThread over an in-memory radio. Without the fix
the stop-window configuration sends only chunks 0x0080 and 0x0100, and the command queued behind an
ACK-first stop is never sent; with it both pass, as does a configuration after the stop has finished.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…res_wrong_instruction

DEV-1306: a stop's reply no longer retires the wrong instruction
DEV-1305: Java HIL rollover, idleStart and samplingRates scenarios
ShimmerPC.deepClone() copies a device by serialising it, which takes no
lock. On a connected device the timers and the IO thread kept changing the
instruction list (and, while streaming, the PC timestamp list and packet
queue) during the copy, so deepClone() could throw
ConcurrentModificationException and the configure built on it failed.

A clone is a copy of the configuration, not of the connection: those three
queues are now transient and readObject() gives the copy empty ones.

The Java HIL's HilShimmerPC had the same problem with its own log, which
the driver's threads append to on every message; its counters are now
transient too.

Tests: API_00034 clones while a thread queues and retires instructions,
and a simulator-connected device while an app reads battery and status;
HilShimmerPCTest clones while the driver logs. All three fail on master.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
On a Shimmer3 streaming at 256 Hz, about one copy in a hundred threw
ConcurrentModificationException in mListofPCTimeStamps, which the IO
thread changes for every packet. And the processing thread's packet queue
holds packets that can't be serialised, so deepClone() returned null
whenever one was waiting. Both tests fail on master.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
…lists

DEV-1310: deepClone() no longer copies the connection's queues

@MAzalya MAzalya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 03106bf. This looks good to me.

  • Lifecycle. The clock starts lazily from the data thread, throttled to one request per 100 ms, and only when it is enabled, the X axis is time and the chart is showing. Each tick stops it on: no sample for 2 s, pause, hidden or iconified, a replaced chart, a non-time X axis, cleared data, or the clock being disabled. removeAllSignals() (called from frameClosed()) stops it immediately. A stopped javax.swing.Timer leaves the TimerQueue, so there is no leak and no cost when idle. Chart2D's repaint latency is restored on stop.
  • Locking with #280. The tick takes only the chart monitor. noteLiveSampleForFrameClock runs under mListofPropertiestoPlot but only calls invokeLater, so the list → chart lock order can't be inverted.
  • Callers that don't enable the clock see no behaviour change. DEFAULT_LINE_THICKNESS is still 2.
  • setTraceLineThickness applies to traces created afterwards, and the javadoc documents that it does not restyle existing traces. That is fine for Advance-API#494, which sets it before any trace exists.

Minor, not blocking:

  • Margin vs post-stall lag. LIVE_VIEWPORT_TRACE_MARGIN_MS (400 ms) is smaller than the lag the clock allows after a data stall. During a stall the window freezes at the newest X. When the buffered data lands below the 1000 ms resync threshold, the window catches up at ≤2× rather than snapping. So a Bluetooth stall longer than 400 ms leaves a blank strip at the left edge, of up to about (stall − 400 ms), for under 0.5 s. A margin of about 1 s would cover it. The dataStall_ test only asserts recovery time.
  • mIsPlotPaused is not volatile but is read by the EDT tick. If pause is set off the EDT, the window can scroll for up to ~200 ms longer before stopping. Benign, but volatile is cheap.

CI: no checks have run on this PR, because the workflow only triggers for PRs into master. So API_00029_LiveViewportFrameClockTest has not run anywhere yet.

Merge order: once #280 merges, retarget this to master and let CI run before merging. If #280 is squash-merged, rebase this branch first, or #280's original commits will reappear in the diff. Advance-API#494 depends on this PR and can't compile in CI until it is on master.

Also: the hardware result in the description (#280 on its own gave no improvement, and the stutter is frame cadence) means #280's description still says "Fixes the stuttering…". It's worth correcting there so the release notes credit the right change.

…pause flag

LIVE_VIEWPORT_TRACE_MARGIN_MS was 2 x VIEWPORT_LATENCY_MS (400 ms), but
after a data stall shorter than FRAME_CLOCK_RESYNC_MS the window catches up
at <=2x instead of snapping, trailing the newest sample by up to
FRAME_CLOCK_RESYNC_MS + VIEWPORT_LATENCY_MS. A Bluetooth stall over ~400 ms
therefore left a blank strip at the window's left edge while it caught up.
The margin is now FRAME_CLOCK_RESYNC_MS + VIEWPORT_LATENCY_MS (1.2 s). New
test dataStall_lagStaysWithinTraceMargin_forStallsUpToResync covers stalls
of 300-980 ms and fails with the old margin.

mIsPlotPaused is now volatile: the frame clock tick reads it on the EDT and
the data thread reads it too, while pause can be set from another thread.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
@MAzalya

MAzalya commented Oct 23, 2026

Copy link
Copy Markdown
Contributor

Addressed in 002b674:

  • Trace margin. LIVE_VIEWPORT_TRACE_MARGIN_MS is now FRAME_CLOCK_RESYNC_MS + VIEWPORT_LATENCY_MS (1.2 s). That is the worst-case lag while the window catches up at <=2x after a stall just short of the resync threshold. New test dataStall_lagStaysWithinTraceMargin_forStallsUpToResync covers stalls of 300, 600, 900 and 980 ms, and it fails with the old 400 ms margin.
  • mIsPlotPaused is now volatile.

All 8 API_00029 tests pass locally on JDK 11. Advance-API#494 picks the new margin up through getLiveViewportTraceMarginInSeconds(), so it needs no code change. Its description is updated, along with this one and #280's, which no longer claims #280 alone fixes the stutter.

@MAzalya

MAzalya commented Oct 27, 2026

Copy link
Copy Markdown
Contributor

not really seen improvement, closing this PR

@MAzalya MAzalya closed this Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants